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1.  Introduction 

Let  f  be  a  real-valued  function  defined  on  the  state  space  of  a 
regenerative  process  X  ■  (X(t)  :  t  _>  0)  with  regeneration  times 
0  ■  Tp  <  Tj  <  •  ••  ,  and  suppose  that 


(1.1) 


.  t 

—  /  f (X(s) )ds  ♦  r 
C  0 


3  •  S  • 


as  t  ♦  ®.  The  problem  of  estimating  r  via  a  simulation  of  X  is  called 
the  steady  state  simulation  problem. 

Relation  (1.1)  implies  that  rc  is  a  strongly  consistent  point  esti¬ 
mator  for  r.  To  obtain  confidence  intervals  for  r,  set  (for  k  >  1) 


Vf) 


T- 


■  T.  - 


f (X(s) )ds 

T.  . 
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The  regenerative  structure  of  X  guarantees  that  r  -  E{Yj(f ) }/E{vl }  and 
that  {(Y^Cf),  :  k  >  1}  is  a  sequence  of  1.1. d.  random  vectors. 

Standard  arguaents  (see  CRANE  and  IGLEHART  (1975))  shov  that  if 
E{Y*(f)  +  xf)  <  -,  then 

(1.2)  /t(rfc  -  r)  ^  oN(0,l) 

2  2 

as  t  +  «,  where  o  •  o  (2^}/E{t^>.  To  use  (1.2)  for  confidence  Inter¬ 
vals,  the  regenerative  cycle  structure  of  X  is  exploited  to  obtain  a 

2 

strongly  consistent  estimator  v£  for  a  . 

These  confidence  Intervals,  while  asymptotically  correct,  often  have 
poor  small-sample  behavior.  For  example,  such  confidence  Intervals  often 
tend  to  significantly  undercover  the  parameter  r.  Several  recent  studies 
have  examined  this  problem.  GLYNN  (1982),  in  considering  regenerative 
confidence  Intervals  on  the  time  scale  of  regenerative  cycles,  obtained 
asymptotic  expansions  for  the  coverage  error  which  Indicated  that  skewness/ 
kurtosls  effects  play  a  significant  role  in  determining  quality  of  the 
confidence  interval.  To  be  more  precise,  the  error,  to  a  first  approxima¬ 
tion,  is  determined  by  the  magnitude  of  quantities  of  the  form  E{z“  t"> 
for  i  +  q  <  L  GLYNN  and  IGLEHART  (1984)  obtain  expressions  for  the 
asymptotic  covariance  between  r£  and  vt>  and  the  variance  of  vf;  these 
expressions  also  Involve  mixed  moments  of  the  form  E{Z™  r”}.  Conse¬ 
quently,  in  studying  small-sample  behavior  of  regenerative  confidence 
Intervals,  it  is  of  some  interest  to  be  able  to  calculate  the  exact  values 
of  the  mixed  moments  for  some  "test  case”  stochastic  models.  HORDIJK, 


IGLEHART ,  and  SCHASSBERGER  (1976)  showed  how  to  do  this  for  a  +  n  <  2, 
when  X  is  a  discrete  or  continuous  tiae  Markov  chain  with  countably  aany 
states.  In  this  note,  we  show  how  to  calculate  such  quantities  when  X  is 
a  8 eai -Markov  process  with  countably  aany  states;  discrete  and  continuous 
tiae  Markov  chain  results  follow  as  special  cases. 

2.  Statcaant  of  tiae  Recursive  Ho went  Formulas 

Let  X  "  (X(t)  :  t  _>  0}  be  an  irreducible  non-explosive  regenerative 
semi-Markov  process  on  countable  state  space  E.  Thus,  X(t)  may  be 
represented  as 

«‘> '  k]0  \  1(sk  i '  <  W  • 

where: 

(i)  1*  (R  :  n  >  0}  is  a  discrete-time  Markov  chain  on  E  with 

n  — 

transition  matrix  P  ■  (p„  :  x,y  e  E) 

xy 

(11)  S  ■  {SQ  :  n  0)  is  an  increasing  sequence  of  jump  times  with 

Sq  *  0  and  differences  -  Sn  which  are  conditionally 

Independent  r.v. 's  given  X. 

The  conditional  distribution  of  a  is  given  by  F(R  ,R  . .  ,dt)  » 

n  n  nti 

P{a  €  dt|K>,  where  F(x,y,0)  ■  0  for  all  x,y  e  E.  Note  that  S  ♦  • 
n  n 

a.s.,  since  X  is  non-explosive  by  assumption.  Fix  z  e  E  as  the 
regenerative  state;  let  T(z)  -  inf{t  >  0  :  X(t-)  t  z,  X(t)  -  z)  and  set 


Y(u)  -  J  u(X(t) )dt 
0 


where  u  :  E  k  is  an  arbitrary  function.  We  wish  to  study  mixed  moments 
of  the  form 


aij(x)  “  Ex{Y<g)1  Y(h)J} 


for  xeE,  0  <  i  <  m,  and  0  £  j  n,  when  g  and  h  are  fixed  func¬ 
tions,  and  m  and  n  are  non-negative  integers.  Throughout  the  paper  we 
shall  use  • }  and  to  denote  conditional  probabilities  and 

expectations,  given  X(0)  ■  ■  x.  Note  that  by  choosing  g(«)  ■  f( •)  -  r 

and  h(«)  “  1,  a^Cz)  yields  E^Z*  tj>. 

To  state  our  result,  let  a  v  b  denote  max(a,b)  and  set  b  (x)  - 

on 

ExUY(|g|)  v  l)“  (Y(  | h | )  v  l)n>.  Let  Gn  and  ?n  be  the  matrix  and 
function,  respectively,  defined  by 


n  (o  ;  y  ■  z 


“  l  >in<X*y)  P, 


where  pn(x,y)  •  tn  F(x,y,dt).  Also,  we  shall  Identify  real-valued 
functions  u(<)  on  E  with  column  vectors  m,  and  shall  use  the  notation 
a  o  w  to  denote  the  vector  with  xtn  component  (s  o  v)(x)  -  u(x)v(x). 


_  _  0,  x  .  .  n+1 

Set  •  (•)■!  and  a 


(u  °  m  )  for  n  >  0. 


Let  C  denote  the  class  of  all  m  *  n  matrix-valued  functions,  C, 


on  E.  Then  set 


•  X  G1+jWx*y)lckJi(y)f  <“ 

y«E  J 

and  (Gq  cij)(x)  +0  as  k  *  •  , 

for  all  x  e  E,  0  £  i  <  m,  and  0  £  j  <  n}  . 


(2.1)  Theorem.  If  b  (*)<*»,  then  A  ■  {a. .  :0<i<a,  0  <  j  <  n} 

on  lj  —  —  —  — 

Is  the  unique  solution  In  C  to  the  system: 

an 


(2.2)  ctj 


+  X 

(k,*)«B 


(i>(i>(*1"k  °  fc3‘1 0  ci+J'k'\i) 


where  0  <  i  <  m,  0  <  j  <  n,  and  -  {(k,l)  :  0  <  k  <  i,  0  <  *  <  j, 

k+A  >  0}. 

Set  x  »  inf (n  >  1  :  R  -  z)  and  observe  that 


^(x.y)  -  Pj{{Rk  «  y,  x  >  k}  . 

Hence,  -*•  0  as  k  ♦  •.  It  follows  that  If  E  has  a  finite  number  of 

elements ,  C 3  C  ,  so  that  the  a  '  s  are  unique  in  the  class  of  all 

mn  ij 

possible  solutions  to  (2.2).  Also,  in  the  presence  of  a  finite  state 
space,  it  is  wall  known  that  (I  -  Gg)  1  exists,  so  that  (2.2)  may  be 


re-written  as 


(2.3) 


eu  * (I  -  V  <*  °  ^  °  *i>j 


♦  i  <l><iH«l-k  o  o  (C  k.Al»|  , 
(k.DeA^ 


where  0  <  i  <  a,  0  <  j  <  n,  i  +  j  >  0,  end  Afj  -  Uk,I)  :  0  <  k  <  i, 

0  <  i  <  J,  0  <  k+A  <  1+j).  Also  observe  that  CqQ  =  1.  Note  that  the 
system  of  equations  (2.3)  is  recursive  in  i+j ,  in  the  sense  that  the 
Cjj's  nay  be  solved  in  terns  of  the  e^'s,  wh«r«  k+A  <  i+j*  By 
successively  solving  for  the  c^'s  with  fixed  k+A  on  each  iteration,  one 

eventually  obtains  c  . 

nn 

Formula  (2.3)  can  be  further  sinplified  when  Z  has  special  struc¬ 
ture.  Note  that  if  Z  is  a  continuous  time  Markov  chain,  then 


F(x,y,dt)  -  X(x)  exp(-X(x)t)dt 


for  t  >  0,  so  that 


u  (x,y)  -  n!/X(x)n  =  r\  (x)  . 
n  n 


We  find  that  (2.3)  can  be  re-written  as 


-  (i  -  cnrv  °  ^ 


u 


°  * 


i+j 


+  .  I  °  **  °  nt+j-k-A  °  C0  ®kA^  ^  * 

(k.DeA^ 


where  0  £  i  <  n,  0  £  j  £  n,  and  i+j  >  0. 

For  discrete  time  Markov  chains,  B  =  1  and  G  »  Gn,  so  (2.3) 

n  n  u 


takes  the  form 


lli  ' 
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el )  ■  (I  -  V'V  0  ^ 


*.  .1  .  Wl"Ht‘H,,C0  <W>t  • 


(k,A)«A 


where  0£l_<m,  0£j£m,  and  i+j  >  0. 

Relation  (2.4)  expresses  in  terms  of  Gq  e^,  where  (k,Jl)  e  Aj 

Equation  (2.4)  can  be  re-written,  when  g  ■  h,  so  that  the  c^’s  are 
written  directly  in  terms  of  the  e^'s.  If  g  ■  h,  we  write  as 

c1+j  .  and  observe  that  (2.4)  takes  Che  form 


(2.5)  et  -  (I  -  C,,)’1!!1  +  ^  (J)(gK  o  CQ  c^)},  l<i<n 


Recall  also  that  from  (2.2),  en  =  1.  We  claim  that  the  system  (2.5)  can 


be  re-written  as 


(2.6)  ct  -  (I  -  gk  °  C!-k}.  1  <  i  <  n 


The  proof  is  by  induction.  For  n  ■  i,  the  result  is  obvious,  so  suppose 
(2.5)  and  (2.6)  are  equivalent  systems  for  n  ■  m.  To  check  the  (mH)'st 
equation  in  the  (mH)'st  system,  observe  that  a  solution  of  (2.5) 
satisfies 


(2.7) 


Vi  •  <■  -  co>'1i«*fl  ♦  J,  <TK«' 


m+1  w  mH-i 


C0  c±) } 


By  the  inductive  hypothesis,  (2.6)  shows  that 


^ 


(2.8) 


“0  i 


-  I 

k-0 


(-1)  k+1(i)gk  o 


i-k 


for  i  <  m.  Substituting  (2.8)  into  (2.7),  we  get  that  (I  -  go^Cb+-i 
equals 


m+1  ?  r  ,«Hwi.,  ,vi-k  aH-k 

g  +  1  1  (  i  )(k)(-D  g  °  e. 

i-1  k-0 

■  *  o  <■  >  *r  (*f!'k)(-i)*fi'k'j 

k-0  *  J-l  J 


-  I  ( 

k-0 

nrt-1 


“  I  ( 
1-1 


nrfl 

k 


arH 

l 


)(g^‘-k  O  %)(-l)*-2-k 

«**  'Vi-iK'1,w  * 


which  is  equivalent  to  the  (aH)'st  relation  of  (2.6)  (the  binomial 
identity  was  used  for  the  third  equality).  The  steps  being  reversible, 
this  proves  the  claimed  result.  We  remark  that  (2.6)  yields  the  equations 
of  [4]  for  i  -  1,  2. 


3.  Proof  of  the  Theorem 

We  proceed  via  a  series  of  lemmas. 

(3.1)  Leonrn.  If  b  (z)  <®,  then  the  r.v. 's  Y(g)1  Y(h)^  are  integrable 

on 

under  the  probability  distribution  P  for  0  £  i  £  a,  0  £  j  £  n,  xeE. 


Proof.  Note  Chat 


<Y(|gl>  v  l)1  (y(]h|  v  1)J  <  (Y( | g )  v  l)m  (Y( |h|  v  l)n 

so  that  b^j(z)  <  *  for  O^i^n,  0  ^  j  £  n.  Since  X  is  irreducible, 

it  follows  that  P  {T(y)  <  T(z)}  >  0  for  all  y  e  E,  for  otherwise  the 
z 

regenerative  property  guarantees  that  ~  P  {T(y>  ■  *}  -  1,  which  violates  our 

z 

irreducibility  assumption. 

Then,  by  the  strong  Markov  property  applied  at  time  T(y) , 

T(z)  .  T(z)  . 

E  f(/  Jg(X(s) Jds  v  l)1  (/  ]h(X(s))]ds  v  l)J;  T(y)  <  T(z)} 

2  T(y)  T(y) 

-  btj(y)  P{T(y)  <  T(z)|X(0)  -  z>  <  b^Cz) 

so  that  <  •  for  all  y  €  E,  which  proves  Che  result.  □ 

(3.2)  Proposition.  If  b  (z)  <®,  then  a,  .(x)  exists  and  is  finite  for 

mn  l  j 

0  <  i  <  a,  0  j  ^  n,  x  «  E.  Furthermore,  A  solves  (2.2). 

Proof.  The  first  part  follows  immediately  from  Lemma  3.1.  For  the  second 
part,  the  integrability  of  Y(g)1,  Y(h)^  ensures  that  the  following  manipu¬ 
lations  of  conditional  expectations  are  valid: 


a^x)  -  Ex{Y(gr  Y(h)J> 


-  E^YCg)1  Y(h)J;  x  -  1}  +  Ex(Y(g)i  Y(h)J;  x  >  1} 

-  g1(x)  h3(x)  ki1+j(x,z)  pxz 

T(z)  , 

+  Ex{(g(R0)  <x0  +  -/  g<X<t»dt>x  (h(Rg)  Oq 
S1 

T(z)  . 

+  /  h(X(t))dt)J;  x  >  1} 

S1 

-  g*(x)  hJ(x)  ni+.j(x,z)  pxz 

+  l  1  <£)<][)  Ex{g(R0)i_k  hCRjj)^-^  aJ+j'lt"A(/  (  }  g(X( t) )dt )k 

k-o  A-0 

T  f 
•  (/  h(X(t))dt)  ;  t  >  1} 

S1 

“  g1(x)  hJ(x)  Ji1+j(x,z)  pxz 

+  I  i  <£><1>  g1_k(x)  h(x)J_1  l  p  n.+.^/x.y) 
k*0  Jt-0  k  *  y>*z  **  1  J  *  1  <JL 

-  (g1  o  h1  o  p  )(x)  +  l  <*)<J1X«i“k  o  fc3'* 

1  J  (k.DfiBjj  * 

°  (Ci+j-k-i  *kJl)(x))  ; 


Che  strong  Markov  property  at  time  S j  was  used  to  obtain  Che  second  last 
equality.  0 


(3.3) 


If  b  (z)  <•,  then  A  e  C  . 


"‘-I 


l 


Proof.  For  the  absolute  summability  observe  that  d^(x) 


■jWlslJ1  Y<  )hj  )J  >  satisfies 

dLj  -  l*)1  °  ]*»]J  °  P1+j 

+ i  „  <i><{)(i*i1‘k » wJ'*  •  (Vj-k.,  -k,))  • 

v  K  9  X/ 

But  d^x)  <  b^x)  <  «,  so  jg]1  k  o  |h|^  1  o  (G1+J_k_A  d kA)  is  finite, 

proving  the  first  part,  since  1*^1  £  ^ j  •  For  the  second, 

T(z)  T(z) 

(Gj  atj)(x)  -  Ex{(/  Y(g))1  (J  Y(h)V;  t  >  k}  , 

Sk  Sk 

which  tends  to  zero  by  integrability  of  Y(jgl)1'  Y(]h|)^. 


(3.4) 


.  If  b  (z)  <  <»,  then  A  is  the  unique  solution  to  (2.2)  in 
mn 


Proof.  Suppose  that  {a  :  0  £  r  <  k,  0  <  s  <  1}  is  unique  in  £.  , 

for  k+f  <  i+j ,  where  0  <  i  <!  m,  0  _<  j  £  n.  We  shall  prove  uniqueness  in 

"bootstrapping"  this  result  yields  the  lemma.  Since  the  a  's  are 
i  j  rs 

unique  in  Cut*  any  solution  to  the  (i,j)  equation  must  satisfy 


(3.5) 


eu  •  «*  °  ^  °  »itj +  (k  j,  <  i)(«>(*,'k  0  ‘i~*  °  ei+j-k-»  \»> 


*  e0  clj  • 


—  ^  *-  -  -*■  - _ .^1 _ *_a  _a  _#  j  j  • 


•-  ••  S  ■>-  v'  ■ 


•.  \  .  kV.A 


By  Che  absolute  suaaability,  any  two  solutions  c^j ,  of  (3.5)  must 

satisfy  (c^  “  cij)  "  G0^cij  ”  clj^‘  Since  ^^cij  "  cij^  >  0  88 

k  ♦  • ,  it  follows  that  j  ■  j ,  proving  uniqueness  in  C  ^ j .  0 

These  above  results  prove  all  the  assertions  of  the  theorem. 
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ABSTRACT:  Let  -X  «  {x(t)A  :  t  >.  0}  be  a  regenerative  semi-Markov  process 
with  countable  state  space  E  and  f  a  real-valued  function  on  E. 
Denote  by  Y.(f)  the  area  under  the  function  f(X(»))  in  the  first 
regenerative  cycle,  ii This  paper  gives  a  recursive  method  for  computing 
moments. -of  the  form  E{ Y™( f )y" (g)}  for  arbitrary  f  and  g,  and 
m,  n  >  1.  A  These  moments  are  needed  to  improve  the  accuracy  of  confidence 
intervals  for  steady  state  parameters  obtained  when  using  the  method  of 
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